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ABSTRACT 

Given  two  pictures  of  a  scene  taken  by  different 
sensors  or  at  different  times,  one  way  of  matching  the 
two  pictures  is  to  extract  a  set  of  distinctive  local 
features  from  each,  and  then  match  the  resulting  point 
patterns.  This  paper  investigates  the  sensitivity  of 
point  pattern  matching  to  various  types  of  noise  and 
distortion,  including  omissions  and  additions,  random 
walks,  rotation  and  rescaling,  as  well  as  the  use  of 
different  feature  detection  operators  to  extract  the 
points.  The  effects  of  additional  information  (e.g., 
feature  types)  in  overcoming  the  effects  of  noise  is 
also  studied. 
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1. 


Introduction 


Matching  two  pictures  of  the  same  scene  is  a  common  problem 
in  computer  vision  and  image  processing.  This  problem  arises  in 
connection  with  registering  pictures  obtained  by  different 
sensors,  or  by  the  same  sensor  at  different  times.  In  such 
situations,  the  occurrence  of  systematic  gray  level  differences 
between  the  pictures,  as  well  as  geometrical  distortions,  often 
makes  it  impractical  to  use  conventional  correlation  techniques 
for  matching.  A  possible  alternative  is  to  segment  the  two 
pictures  into  regions,  and  attempt  to  pair  off  the  corresponding 
regions  based  on  their  properties  (color,  size,  shape,  etc.) 

[1] ;  but  this  approach  depends  on  the  reliability  of  the  seg¬ 
mentation  process. 

This  paper  explores  another  approach,  based  on  extracting 
a  set  of  local  features  from  each  picture,  and  then  matching 
the  resulting  point  patterns  (of  local  feature  positions) .  We 
will  not  discuss  the  nature  of  the  local  features;  they  might, 
for  example,  be  corners  on  a  contour  or  branch  points  on  a 
curve.  We  do  assume  that  the  local  feature  extraction  process 
will  be  relatively  stable  under  the  changes  that  take  place 
from  one  picture  to  another.  In  any  case,  it  will  be  seen  that 
the  matching  process  is  relatively  insensitive  to  the  addition 
and  deletion  of  feature  points.  This  paper  is  an  abridgement 
of  a  more  detailed  report  [2],  which  is  the  first  author's  M.S. 
thesis;  it  also  includes  some  supplemental  results  that  were 


subsequently  obtained  by  the  third  author,  who  acknowledges 
the  guidance  of  Charles  R.  Dyer  in  carrying  out  this  work. 

In  eai  ier  related  work,  Simon  et  al.  13)  investigates 
the  matching  of  point  patterns  by  matching  sets  of  interpoint 
distances;  their  method  is  applicable  only  to  patterns  that 
contain  equal  numbers  of  points.  Zahn  )4]  matches  point  pat¬ 
terns  by  comparing  their  minimal  spanning  trees;  this  approach 
is  sensitive  to  point  additions  and  deletions  that  change  the 
structure  of  the  trees.  Seidl  l 51  proposes  measures  of  point 
pattern  similarity  based  on  mutual  nearest  neighbor  relations, 
but  does  not  show  how  useful  these  measures  might  be  in  prac¬ 
tice  when  matching  noisy  or  distorted  patterns. 

Section  2  of  this  paper  describes  the  basic  matching  algo¬ 
rithm  and  gives  examples  of  its  use.  Section  3  studies  the 
effects  of  add  it  ion/de  let.  ion  and  random  walk  noise,  as  well  as 
rotation  and  rescaling,  on  the  matching  process.  Section  4  deal 
with  the  use  of  feature  labels,  associated  with  the  points, 
to  reduce  noise  effects.  Section  5  discusses  the  results  and 
suggests  directions  for  further  work. 


2.  Point  pattern  matching 

The  basic  point  pattern  matching  algorithm  operates  as 
follows:  Suppose  that  we  are  given  two  point  patterns 

n:Pj,...,Pm  and  as  well  as  a  match  tolerance  t. 

We  want  to  find  a  translation  that  maps  11  onto  as  well  as 
possible.  Let  T  be  the  set  of  possible  translations  (we 
assume  the  translations  to  be  quantized,  so  that  T  is  a  finite 
set).  For  each  pair  of  P's  (P.,P^)  and  each  pair  of  Q's 
(0^,0^),  we  check  whether  the  vector  difference  P.P^-O^O^  has 
magnitude  less  than  t;  if  so,  the  translation  that  maps  P. 
onto  Qh  (and  hence,  approximately ,  p.  onto  0^.)  is  a  possible 
mapping  of  11  onto  .  ,  and  we  increment  the  merit  of  this  trans¬ 
lation  by  1.  This  process  is  repeated  for  each  pair  of  pairs. 

If  II  is  reasonably  similar  to  ,  this  should  result  in  a  clustei 
of  high  merit  values  corresponding  to  t ransl at  ion (s)  that  map 
II  onto 

The  following  are  some  comments  on  the  relationship  between 
this  algorithm  and  a  correlation  approach.  Ordinary  cross- 
correlation  (shift,  pointwise  multiply,  and  sum)  would  not  be 
very  useful  for  matching  point  patterns  in  the  presence  of  dis¬ 
tortion,  since  most  of  the  points  will  not  match  up  exactly  ami 
so  will  not  contribute  to  the  sum .  (This  assumes  that  the 
points  are  l's  on  a  background  of  0's.)  However,  we  can  intro¬ 
duce  a  tolerance  into  the  correlation  process  by  "blurring"  one 
set  of  points  into  disks;  when  the  other  points,  after  shifting, 
fall  on  these  disks,  they  do  contribute  to  the  sum.  Our 


algorithm  is  roughly  equivalent  to  correlating  one  pattern 
(unblurred)  with  a  blurred  version  of  the  other  pattern; 
the  score  for  a  given  displacement  is  related  to  the  number 
of  displaced  points  that  fall  on  disks.  However,  our  algo¬ 
rithm  may  be  computationally  cheaper  if  the  patterns  are  very 
sparse,  since  the  cost  of  correlation  goes  up  with  the  array 
size,  whereas  our  algorithm  depends  only  on  the  number  of 
points. 

Several  examples  of  the  use  of  this  algorithm  will  now  be 
given.  Figure  1  shows  a  picture  of  an  armored  tank 
on  which  two  people  had  independently  selected  a  set  of  20 
or  fewer  significant  feature  points  (indicated  by  circles) ; 
note  that  these  sets  have  only  about  half  the  points  in  common. 
Figure  2  shows  the  array  of  merit  values  corresponding  to 
relative  translations  of  these  two  point  patterns.  As  can  be 
seen,  there  is  a  major  peak  indicating  the  translation  for 
which  the  best  match  occurs.  Figures  3  and  4  show  similar 
results  for  a  roadmap  of  the  Washington,  D.C.  area.  In  all 
these  examples,  the  point  positions  are  coarsely  quantized 
using  a  32x32  grid. 

The  tolerance  t  used  in  our  experiments  was  taken  to  bo 
a  specified  fraction  of  the  point  pair  displacements  (i.e., 
of  min  (P^Pj,  Q^Q^)).  In  Figure  2,  t  was  10%;  Fiuure  4a-d 
shows  the  results  of  using  t=0,  5%,  10%,  and  15%.  In  the 
subsequent  experiments  we  used  t=5%. 


Figure  5  shows  results  for  three  FLIR  images  of  a  tank, 
selected  from  a  sequence  of  frames.  Feature  points  were 
automatically  selected  by  applying  an  edge  detector,  based 
on  differences  of  averages  taken  over  4-by-4  blocks  of  pixels, 
suppressing  nonmaxima,  and  thresholding  to  retain  the  best 
30  to  35  points.  Arrays  of  merit  values  are  shown  for  com¬ 
parisons  of  frames  1  and  5,  5  and  10,  and  1  and  10.  They  show 
strong  clustering  corresponding  to  the  motion  of  the  tank  from 
frame  to  frame. 

Figures  6  and  7  show  the  results  of  an  analogous  experiment 
using  two  different  feature  detectors  (namely,  edge  detectors 
based  on  differences  of  2x2  averages  and  of  4x4  averages)  for 
FLIR  images  of  a  tank  and  of  an  APC.  Here  again,  strong  clust¬ 
ering  is  apparent.  The  weaker  secondary  responses  in  Figures 
5-7  represent  matches  between  one  edge  of  the  tank  (or  APC) 
and  the  other;  the  smearing  out  of  the  responses  represents 
matches  between  different  parts  of  the  same  edges. 


3. 


Noise  effects 


The  matching  process  is  evidently  relatively  immune  to 
random  addition  and  deletion  of  points,  since  it  yields  good 
match  peaks  even  on  patterns  that  have  only  half  of  their 
points  in  common.  It  is  sensitive,  however,  to  random  dis¬ 
placements  of  the  points,  if  these  usually  exceed  the  match 
tolerance.  To  illustrate  this,  Figure  8  shows  the  effects 
of  applying  various  amounts  of  random  jump  noise  to  the  first 
set  of  map  feature  points  in  Figure  3.  The  jump  for  each 
point  was  independently  chosen,  with  direction  uniformly  dis¬ 
tributed  in  (  0 , 2 tt  ]  ,  and  with  magnitude  normally  distributed 
with  mean  0  and  standard  deviations  of  about  3*,  4^-%,  6% ,  and 
9%.  The  corresponding  arrays  of  merit  values  are  shown  in 
Figure  9;  all  of  them  display  clustering,  but  it  is  much 
weaker  for  the  higher  noise  magnitudes. 

The  matching  process  is  also  quite  sensitive  to  geometri¬ 
cal  transformations  such  as  rotation  and  scale  change,  since 
it  assumes  that  the  patterns  differ  only  by  translation. 
Figure  10  shows  merit  arrays  for  the  other  set  of  map  feature 
points  matched  against  itself  rotated  3°  and  10°;  clustering 
is  detectable  only  for  the  3°  rotation.  Figure  11  shows  ana¬ 
logous  results  for  rescalings  of  2%,  5%,  10%,  and  20%;  the 
first  three  of  these  still  show  clustering. 


.. . o*, .. 


_ _  . 


4.  Labe  lied  point  pattern  matching 

Up  to  now  we  have  considered  only  patterns  of  identical 
points.  This  required  us  to  use  a  relatively  low  match 
tolerance  t=5*,  since  hiqher  t's  would  yield  many  ambiguous 
matches.  As  a  result,  the  matching  process  was  relatively 
sensitive  to  point  displacement  and  other  distortions. 

This  noise  sensitivity  can  be  reduced  by  allowing  the 
points  to  have  labels  (e.g.,  representing  types  of  features). 
This  permits  the  match  tolerance  to  be  increased  without 
greatly  increasing  the  ambiguity,  since  pairs  of  points  having 
the  same  labels  are  less  likely  to  match  by  chance.  To 
illustrate  this,  in  the  road  map  example  the  points  were 
labelled  by  junction  types  as  shown  in  Table  1.  The  match 
tolerance  was  then  increased  to  15*.  This  yielded  useful 
clustering  over  a  wider  range  of  noise  and  distortion  condi¬ 
tions.  For  example,  Figure  12  shows  rotation  results  for  5° 
and  10°;  clustering  is  now  apparent  for  both  angles. 


5.  Conclusions 


The  point  pattern  matching  algorithm  described  in  this 
paper  is  very  simplistic;  it  only  considers  points  pairwise, 
and  uses  a  simple  threshold  t  to  decide  whether  or  not  a 
given  pair  of  pairs  match.  Nevertheless,  it  seems  to  yield 
useful  results  in  a  variety  of  cases,  and  should  be  applicable 
to  situations  where  the  pictures  to  be  matched  differ  primarily 
by  a  translation,  and  where  local  features  can  be  detected 
with  reasonable  reliability  in  both  pictures. 

As  a  generalization  of  our  algorithm,  we  could  allow  the 
merit  increments  to  depend  on  the  value  of  the  discrepancy, 
rather  than  simply  using  a  unit  increment  when  the  discrepancy 
is  below  threshold.  This  would  be  roughly  equivalent  to 
"chamfer  matching"  the  two  patterns  [6] ;  compare  the  remarks 
in  Section  2  about  matching  dots  with  disks. 

The  computational  cost  of  the  algorithm  is  relatively 
high,  since  it  compares  all  possible  pairs  of  point  pairs. 

When  labelled  points  are  used,  this  cost  is  greatly  reduced. 

For  feature  points  that  lie  along  borders  or  curves,  one  could 
also  use  their  order  as  a  constraint  on  which  pairs  to  match; 
compare  [7],  Further  reduction  should  be  possible  by  employing 
relaxation-like  techniques  [8]  to  eliminate  possible  pairings 
based  on  local  evidence.  Heuristic  search  techniques  might 
also  be  used  to  find  good  overall  matchings. 


Other  methods  can  bo  used  to  match  point  patterns  that 
differ  by  transformations  other  than  translation.  As  a  simple 
example,  suppose  that  the  patterns  differ  by  rotation  about 
a  known  center.  We  can  histogram  the  set  of  slopes  defined 
by  all  the  point  pairs  in  each  pattern,  and  then  find  the 
cyclic  shift  that  yields  the  best  match  between  the  two 
histograms.  This  idea  is  illustrated  in  Figure  13. 

It  is  well  known  [9]  that  when  matching  pictures  that 
may  differ  substantially  in  grayscale,  it  is  advantageous  to 
match  edge  detector  outputs  for  the  pictures  rather  than  the 
pictures  themselves,  since  the  edge  outputs  represent  the 
geometry  of  the  pictures'  contents  without  being  greatly 
affected  by  the  gray  level  differences.  The  present  approach, 
using  the  outputs  of  local  feature  detectors,  can  be  regarded 
as  a  generalization  of  the  edge-based  approach. 

In  summary,  feature  point  pattern  matching  seems  to 
provide  a  potentially  useful  approach  to  picture  matching 
under  substantial  grayscale  modification  and  moderate  geome¬ 
trical  distortion.  Further  studies  of  this  approach  are 


planned . 
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N umbe r  of 
Branches 

2 

3 

3 

3 

4 

4 

5  or  more 


Type 

Angle  ("L" ) 

"Arrow"  (branches  all 
lie  in  a  half-plane) 

M  rjt  h 

"Fork"  (branches  not 
in  a  half-plane) 

"X"  (branches  collinear) 

Branches  noncollinear 


Table  1.  Junction  types 


Figure;  Captions 


No. 

la-b 

2 

,la-b 

4a-d 

rvi-c 

ha-b 

7a-b 

8a-e 

9a-d 

I0a-b 

11  a-d 

12  a-b 
Ua-d 


Two  sots  of  feature  points  for  tank  picture 
Array  of  match  merit  values  for  tank  feature  points 
Two  sets  of  feature  points  for  map 


Arrays  of  match  merit  values  for  tolerances  of  0*, 
5H,  101#  and  15t  of  image  sire 

Arrays  of  match  merit  values  for  edge  points  ex¬ 
tracted  from  three  infrared  images  of  a  tank: 

(a)  Frames  1  and  5,  < b )  frames  5  and  10, 

(c)  frames  1  and  10. 

a)  Edge  points  extracted  from  an  infrared  imaqe  of 
a  tank  using  two  different  edge  detectors 

b)  Array  of  match  merit  values  for  these  two  sets 
of  points 

Analogous  to  Figure  7  for  an  image  of  an  APC 

Results  of  applying  random  jump  noise  to  the  map 
feature  points  in  Figure  la.  (a)  Original 
points;  (b)  o=  31;  (c)  >’  =  4-1/21;  (d)  o  =  6%; 
(e)  o  =  9* 

Arrays  of  match  merit  values  corresponding  to 
Figures  9b-e.  ' 

Arrays  of  match  merit  values  foi  the  map  feature 
points  in  Figure  3b  under  (a)  5*, and  (b)  10*’ 
rotation 

Analogous  to  Figure  11  for  rescalings  of  (a)  21, 

(b)  51,  (c)  10*,  and  (  d)  20*. 

Analogous  to  Figure  11  using  labelled  feature 
points  and  a  tolerance  of  1 5 » . 

Numbers  of  slope  matches  as  a  function  of  rotation 
angle  for  pairs  of  point  patterns  that  differ 
by  rotations  of  (a)  5°,  (b)  10°,  (c)  15°,  (d) 

(d)  20°. 
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Figure  10b 
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Figure  13.  Rotational  histograms  for  rotations  of  (a)  5°,  (b)  10°, 
(c)  15°,  (d)  20°.  Note  that  the  histogram  peak  (underlined)  in  each 
case  is  in  the  range  corresponding  to  the  actual  rotation. 
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